Hierarchical Attribute CNNs

Deep neural network algorithms are difficult to analyze because they lack structure allowing to understand the properties of underlying transforms and invariants. Multiscale Hierarchical Convolutional Networks are a theoretical class of structured deep convolutional networks that constitute a framework to understand neural network classification properties. However, a naive implementation of such networks is infeasible due to the exponential growth of parameters with depth, so their empirical properties remain to be studied. We introduce a subclass of them that overcomes this issue, called Hierarchical Attribute Convolutional Networks, where individual layers are indexed by progressively higherdimensional and increasingly invariant attributes. Each new layer is computed with multidimensional convolutions along spatial and attribute variables. The dimensionality is kept constant by averaging intermediate layers along attributes, allowing to control the size of the layers while reducing their parameters. This permits to train Hierarchical Attribute CNNs on CIFAR image databases where they obtain comparable accuracy to state of the art networks while having much fewer trainable parameters. We study some properties of the attributes learned from Cifar-10.